Motion compensated video spatial up-conversion

ABSTRACT

A method for performing motion compensated video spatial up-conversion on video. The horizontal samples in successive fields are first interpolated using a spatial interpolation technique. This is followed by interpolating the corresponding vertical samples using a motion compensated deinterlacing technique. Such techniques can include an adaptively recursive motion compensated video spatial up-conversion or an adaptively recursive motion compensated video spatial up-conversion using a generalized sampling theorem. The present invention can be used to convert video captured on a mobile device, such as a mobile telephone, so that it can be subsequently and adequately displayed on a television.

FIELD OF THE INVENTION

The present invention relates generally to video processing. Moreparticularly, the present invention relates to video spatial-upconversion using motion compensation in video processing.

BACKGROUND OF THE INVENTION

Video spatial up-conversion (V-SUC) is also known as video resolutionenhancement. V-SUC is used to enhance the spatial resolution of anarbitrary video sequence through both horizontal and vertical spatialinterpolation. Video spatial up-conversion is one aspect of video formatconversion (VFC), in which video signals are converted from one formatto another. Two typical aspects of VFC are video deinterlacing, alsoknown as video scan rate up-conversion and video picture rateup-conversion. Deinterlacing involves enhancing the spatial resolutionof a video signal through interpolation in the vertical direction. Videopicture rate up-conversion enhances the picture rate (also known asframe rate) of a video signal through temporal interpolation.

Video spatial up-conversion is required for TV-out of mobile phonecaptured videos. Typical spatial resolutions of NTSC TV are 640×480 or800×576. In contrast, videos captured by conventional mobile telephoneshave a spatial resolution typically as SIF (320×240), CIF (352×288), orQCIF (176×144). Therefore, the spatial resolution needs to be enhancedbefore mobile telephone-captured videos are displayed in a regular TVdevice. Another example of video spatial up-conversion involves thedisplay of standard definition TV (SDTV) signals in a high definition TV(HDTV) device.

Video spatial up-conversion mainly needs to fulfill two tasks in theprocess of spatial resolution enhancement: anti-aliasing and highspatial frequency generation to overcome the over-smoothness artifact.

A digital video signal is obtained through three-dimensional (3D)sampling of the original continuous video signal. For example, Δx, Δy,and T can denote the sampling distances in the horizontal direction, thevertical direction, and the temporal direction, respectively, whichspecify a 3D sampling grid. In this situation the Fourier spectrum ofthe digital video signal is the ensemble of multiple replications of theFourier spectrum of the continuous video signal along the 3D samplinggrid that is specified by the sampling frequencies, f_(s) ^(x), f_(s)^(y), and f_(s) ^(t), where f_(s) ^(x)=1/(Δx), f_(s) ^(y)=1/(Δy), andf_(s) ^(t)=1/T. The replication centered at the coordinates (0,0,0) isreferred to as the baseband spectrum. If the original continuous signalis band-limited and the maximum frequencies in the respectivedirections, denoted as f_(max) ^(x), f_(max) ^(y), and f_(max) ^(t)respectively, satisfy the following constraints, namely f_(max)^(x)≦f_(s) ^(x)/2=1/(2Δx), f_(max) ^(y)≦f_(s) ^(y)/2=1/(2Δy), andf_(max) ^(t)≦f_(s) ^(t)/2=1//(2T), then the continuous signal can becompletely recovered from its 3D samples. Ideal interpolation filteringthen corresponds to all-pass the baseband spectrum and the otherreplications are zeroed-out. If the above constraints are violated, thenadjacent spectral replications will overlap with each other, resultingin aliasing.

When a continuous video signal is sampled, anti-aliasing filtering isfirst applied so that all the frequencies that are larger than half ofthe respective sampling frequency are removed, avoiding the problem ofaliasing. However, this is not the case for progressively scanned videosthat are captured by cameras. It is known that sampling in both thevertical and temporal directions is part of the scanning formatintegrated with the camera. The desired anti-aliasing is thereforerequired in the optical path of the camera, which is extremely difficultand expensive to realize. Therefore, aliasing is usually present in thef_(y)-f_(t) frequency space, as shown in FIG. 1. In the f_(y)-f_(t)frequency space, the extent of the spectrum support is determined by thevertical details of the scene, while the spectrum orientation isdetermined by the vertical motions.

When a digital video signal is upsampled, an ideal interpolation filtershould all-pass the baseband spectrum, without aliasing, whilesuppressing the aliasing portion as much as possible. As shown in FIG.1( b), if a vertical motion is present, an ideal low pass filter forinterpolation should be motion-compensated to effectively extract thebaseband spectrum without aliasing.

In contrast, horizontal sampling is realized after the image acquisitionprocess. For this reason, anti-aliasing filtering can be implemented inthe horizontal direction before sampling. This implies that, for videospatial up-conversion, the interpolation in the horizontal direction andthe vertical direction should be treated separately. Because the highfrequency component is either filtered out in the process of sampling orsuppressed due to aliasing in the process of upsampling, the videosignal after spatial up-conversion is lacking the high frequencycomponent, resulting in the blurring or over-smoothness of artifacts.Many spatial filters have been designed to boost the high frequencycomponent during spatial interpolation.

Conventional techniques for video spatial up-conversion have beenprimarily realized through spatial interpolation in a frame-by-framebasis. For this reason, spatial interpolation techniques for 2D stillimages have been directly extended to the use for video signals, wherecorrelation across different frames of a digital video has beencompletely ignored.

Spatial interpolation using finite impulse response (FIR) filtering isthe most commonly used technique, where image independent FIR filtersare applied in both the horizontal direction and vertical direction of astill image. Various interpolation FIR filters have been designed, withtypical examples as bilinear filter, bicubic filter, bicubic splinefilter, Gaussian filter, and Lanczos filter. These FIR filters aredifferentiated from each other mainly by different passband and stopbandfrequencies, as well as the length of the filter kernels. The design ofthese FIR filters mainly aims to all-pass the baseband spectrumcontaining no alias, suppress the aliasing spectrum component, and boosthigh frequencies to preserve image details such as edges. As wementioned, proper anti-aliasing is usually applied prior to horizontalsampling but not in vertical sampling, it is suggested that differentfilters be used for horizontal interpolation and for verticalinterpolation.

Image content-dependent filters have also been developed for imagespatial interpolation. On such filter is referred to as the Wienerfilter, which is a linear filter with a target at the least mean squareerror (MSE). The coefficients of these types of filters are derived fromthe local image content, thus adapting to the local imagecharacteristics. Other image spatial interpolation techniques are alsoconventionally known. These techniques include New Edge-DirectedInterpolation (NEDI), which uses the geometrical duality acrossdifferent resolutions of the image content, and Adaptive Quadratic(AQua) image interpolation, which is based upon the optimal recoverytheory and can be used to permit the interpolation of images byarbitrary factors. It has been shown that longer FIR filter kernels orimage dependent filters are often preferred.

Nevertheless, for the techniques that use spatial interpolation forvideo spatial up-conversion in a frame-by-frame basis, the correlationalong the motion trajectory in the temporal direction has been widelyignored. It is known that NEDI has been extended for the use of videospatial up-conversion by taking into account of motion compensation.However, this consideration of motion compensation is confined to aspecific schematic framework. Additionally, motion compensation has beenconsidered for “superresolution,” a recently emerged application alsoaiming to enhance the spatial resolution of an arbitrary video signal.However, superresolution is considerably different from video spatialup-conversion in the sense that superresolution is targeted to generateone or a limited set of images from a given video sequence with enhancedspatial resolution. In contrast, video spatial up-conversion aims toenhance the spatial resolution of every picture in the video sequence.An effective video spatial up-conversion technique is only permitted touse a limited number of adjacent frames to enhance the resolution ofcurrent frame and the computational complexity should be kept reasonablylow. Therefore, the concept of motion compensated video spatialup-conversion has not been extensively examined.

SUMMARY OF THE INVENTION

The present invention involves the designing of effective motioncompensated video up-conversion techniques by taking advantage of theconnection between video spatial up-conversion and video deinterlacing.Specifically, the present invention involves the idea that interpolationin the two spatial directions for video spatial up-conversion be treateddifferently, and motion compensated techniques be used for theinterpolation in the vertical direction.

The present invention addresses the two primary tasks involved in theprocess of spatial resolution enhancement for video spatialup-conversion. In particular, the present invention addresses bothanti-aliasing and high spatial frequency generation, which serves toovercome the over-smoothness artifact that would otherwise exist usingconventional approaches. With the present invention, video resolution isenhanced by a scaling parameter of 2 in both the horizontal and thevertical directions.

These and other advantages and features of the invention, together withthe organization and manner of operation thereof, will become apparentfrom the following detailed description when taken in conjunction withthe accompanying drawings, wherein like elements have like numeralsthroughout the several drawings described below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1( a) is a representation of the f_(y)-f_(t) frequency space forprogressively scanned videos with vertical motion, and FIG. 1( b) is arepresentation of the f_(y)-f_(t) frequency space for progressivelyscanned videos without vertical motion;

FIG. 2 is a representation of a three-dimensional sampling grid forvideo deinterlacing;

FIG. 3 is a representation a three-dimensional sampling grid for videospatial up-conversion;

FIG. 4( a) is a representation showing an example of verticalinterpolation using motion compensated samples with video deinterlacing,and FIG. 4( b) is a representation showing an example of verticalinterpolation using motion compensated samples with video spatialup-conversion;

FIG. 5 is a representation of four types of samples in video spatialup-conversion;

FIG. 6 is a representation of motion compensated interpolation using theGeneralized Sampling Theorem (GST);

FIG. 7 is a perspective view of an electronic device that canincorporate the principles of the present invention; and

FIG. 8 is a schematic representation of the circuitry of the electronicdevice of FIG. 7.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the present invention, the close connection between video spatialup-conversion and video deinterlacing is addressed. This is significantbecause, once the resemblance between these two aspects of video formatconversion (VFC) is clarified, the success in the area of videodeinterlacing can be directly migrated to the area of video spatialup-conversion. In particular, motion compensated video up-conversiontechniques can be easily developed by extending the respective motioncompensated video deinterlacing algorithms.

A unique and significant characteristic of video signals is motion.Considering the correlation along the motion trajectory in the task ofvideo spatial resolution enhancement is beneficial. Motion compensatedvideo spatial up-conversion techniques, however, is even moreadvantageous than the consideration of such a correlation without usingtemporal correlations in constructing a spatial resolution enhancedvideo with a superior quality. This fact is supported by the advantagesof motion compensation in video deinterlacing and the close connectionbetween video spatial up-conversion and video deinterlacing.

The present invention involves the concept of designing effective motioncompensated video up-conversion techniques by taking advantage of theconnection between video spatial up-conversion and video deinterlacing.Specifically, the present invention involves the idea that interpolationin the two spatial directions for video spatial up-conversion be treateddifferently, and motion compensated techniques be used for theinterpolation in the vertical direction. Video spatial up-conversion isrequired for TV-out of mobile visual content. With the rapid evolutionin the merging of the mobile visual content service and the traditionalTV business, effective video spatial up-conversion is becoming moredemanding in the consumer electronics market.

For motion compensated video spatial up-conversion, accurate motionvectors are required. It should be noted that the motion for videopredictive coding is different from the motion for video formatconversion (VFC). In video predictive coding, the motion vectors of oneblock do not have to be correlated to that of adjacent blocks. For videoformat conversion, on the other hand, the true motion is supposed to beidentified, where the motion vectors of adjacent blocks that belong toone object should be correlated to each other. Such motions can beobtained for video spatial up-conversion in a similar manner as themotion estimation operation implemented for video deinterlacing or videoframe rate up-conversion. Motion compensated video spatial up-conversiontechniques require more computational resources than non-motioncompensated techniques due to the requirement of motion estimation.However, motion estimators are conventionally known and can be used forvideo deinterlacing. Therefore, the additional cost for motionestimation in video spatial up-conversion is limited.

According to the present invention, video deinterlacing is used toenhance the video vertical resolution, as shown in FIG. 2. In contrast,video spatial up-conversion is used to enhance the video resolution bothhorizontally and vertically. This is represented in FIG. 3.

As depicted in FIG. 3, for video spatial up-conversion, if the originalvideo signal is sampled with spatial distances (Δx′, Δy′), one cantransform the 3D sampling grid to a grid with the same temporal samplingdistance but with spatial sampling distances of Δx=Δx′/2 and Δy=Δy′/2,respectively. If the horizontal resolution is first enhanced, i.e., thehorizontal samples are first interpolated through the use of FIRinterpolation filtering, then the interpolated horizontal samples can berecursively used for the enhancement of the vertical resolution.Compared to the 3D sampling grid for interlaced videos in FIG. 2, theinterpolation of vertical samples for video spatial up-conversion inFIG. 3 can be realized in a similar manner, except that the “original”vertical sampling lines in adjacent frames are located in the samepositions for video spatial up-conversion, instead of being interlacedfor the scenario of video deinterlacing. Therefore, a close connectionbetween video spatial up-conversion and video deinterlacing isestablished from the 3D sampling grid perspective.

Motion compensated deinterlacing techniques take the motion compensatedsamples from the previous frame (or from both the previous and thesuccessive frames) as a candidate for the interpolated samples of thecurrent frame. It should be noted that, although the examples discussedherein refer specifically to frames, a variety of different types offields, which may comprise frames, portions of frames, or othercollections of information, may also be used in conjunction with thepresent invention. Due to the connection between video spatialup-conversion and video deinterlacing, a motion compensated videodeinterlacing technique of the present invention should be able to bemodified and extended for the use of video spatial up-conversion.

The following is a simple example showing the implementation of oneembodiment of the present invention. For video deinterlacing as shown inFIG. 4( a), if a video signal only contains objects with a uniformvertical velocity of v=2kΔy/T, k∈Z, then the interpolated samples can besimply replaced by the motion compensated samples from the previousframe. Analogously, for video spatial up-conversion as shown in FIG. 4(b), if a video signal only contains a vertical velocity of v=(2k+1)Δy/T,k∈Z, then the interpolated samples can also be simply replaced by themotion compensated samples from the previous frame. As the role theoperation of motion compensation plays in video deinterlacing, the useof motion compensation is also beneficial in video spatial up-conversiondue to the consideration of temporal correlation along the motiontrajectory of a video sequence.

Generally, methods for motion compensated video spatial up-conversionaccording to the present invention can be implemented through two steps.The first step involves interpolating the horizontal samples using awide variety of spatial interpolation techniques. The second stepinvolves interpolating the vertical sampling through the use of a motioncompensated deinterlacing-like technique. The use of different methodsfor spatial interpolation in the horizontal direction and in thevertical direction is permissible because horizontal sampling isimplemented after the image acquisition procedure, while verticalsampling is realized as a part of the image acquisition process by thecameras. The success of motion compensation in video deinterlacing, andthe close connection between the two aspects of video format conversion(VFC), imply the success of motion compensation in video spatialup-conversion.

In different embodiments for implementing the process of the presentinvention, two video deinterlacing methods are selected. These methodshave demonstrated particularly strong deinterlaced video quality. Thesemethods are used to develop two motion compensated video spatialup-conversion techniques.

Algorithm I: Adaptively Recursive Motion Compensated Video SpatialUp-Conversion.

Extended from the Adaptive Recursive video deinterlacing technique, thefollowing Adaptive Recursive Motion Compensated (ARMC) video spatialup-conversion technique can be implemented:

$\begin{matrix}{\;{{F_{SUC}( {\overset{arrow}{x},n} )} = \{ \begin{matrix}{{{\alpha_{A}( {\overset{arrow}{x},n} )}{F( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{A}( {\overset{arrow}{x},n} )}} )F_{SUC}}} \\{( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} ),} \\{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {0,0} )} \\{{{\alpha_{B}( {\overset{arrow}{x},n} )}{F_{init}( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{B}( {\overset{arrow}{x},n} )}} )F_{SUC}}} \\{( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} ),} \\{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {1,0} )} \\{{{\alpha_{C}( {\overset{arrow}{x},n} )}{F_{init}( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{C}( {\overset{arrow}{x},n} )}} )F_{SUC}}} \\{( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} ),} \\{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {0,1} )} \\{{{\alpha_{D}( {\overset{arrow}{x},n} )}{F_{init}( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{D}( {\overset{arrow}{x},n} )}} )F_{SUC}}} \\{( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} ),} \\{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {1,1} )}\end{matrix} }} & (1)\end{matrix}$

F({right arrow over (x)},n) denotes the original sample, F_(init)({rightarrow over (x)},n) denotes the initially interpolated sample, andF_(SUC)({right arrow over (x)},n) denotes the ultimately interpolatedsample after video spatial up-conversion, respectively, all at discretespatial coordinates {right arrow over (x)}=(x,y)^(T) and temporalcoordinate n. (·)^(T) denotes the transpose of a vector/matrix. The fourtypes of coordinates in the 3D sampling grid, as shown in FIG. 5, areindicated by “A” for (x mod 2, y mod 2)=(0,0), “B” for (x mod 2, y mod2)=(1,0), “C” for (x mod 2,y mod 2)=(0,1), and “D” for (x mod 2,y mod2)=(1,1). {right arrow over (d)}({right arrow over (x)},n)=(d_(x)({rightarrow over (x)},n), d_(y)({right arrow over (x)},n))^(T) denotes themotion vector of the sample located in ({right arrow over (x)},n).

Any spatial interpolation technique can be used for generating theinitially interpolated samples F_(init)({right arrow over (x)},n) atlocations B, C, and D. Different FIR filters can be selected for theinterpolation of horizontal samples (B) and vertical samples (C and D).

α_(A)({right arrow over (x)},n) is determined by the reliability of themotion vector for the original sample A:

$\begin{matrix}{{{\alpha_{A}( {\overset{arrow}{x},n} )} = {{CLIP}( {0,{c\sqrt{{{{F( {\overset{arrow}{x},n} )} - {F_{SUC}( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} )}}},}1}} )}}{{{where}\mspace{14mu}{{CLIP}( {m_{1},a,m_{2}} )}} = \{ \begin{matrix}a & {m_{1} \leq a \leq m_{2}} & \; \\m_{1} & {a < m_{1}} & {{and}\mspace{14mu} c\mspace{14mu}{is}\mspace{14mu}{{scalar}.}} \\m_{2} & {a > m_{2}} & \;\end{matrix} }} & (2)\end{matrix}$

α_(B)({right arrow over (x)},n) is selected in a way such that thenon-stationary pixels along the motion trajectory for sample B is thesame as that of its horizontally neighboring pixels after video spatialup-conversion:

$\begin{matrix}{{\alpha_{B}( {\overset{arrow}{x},n} )} = {{CLIP}( {0,\frac{{{{\beta_{B\; 1} + \beta_{B\; 2}}}/2} + \delta}{{{{F_{init}( {\overset{arrow}{x},n} )} - {F_{SUC}( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} )}}} + \delta},1} )}} & (3)\end{matrix}$

In Equation (3), δ is a small constant preventing division by zero, and{right arrow over (μ)}_(x)=(1,0)^(T),β_(B1) =|F({right arrow over (x)}−{right arrow over (μ)} _(x) ,n)−F_(SUC)({right arrow over (x)}−{right arrow over (μ)} _(x) −{right arrowover (d)}({right arrow over (x)},n),n−1)|,β_(B2) =|F({right arrow over (x)}+{right arrow over (μ)} _(x) ,n)−F_(SUC)({right arrow over (x)}+{right arrow over (μ)} _(x) −{right arrowover (d)}({right arrow over (x)},n),n−1)|.

α_(C)({right arrow over (x)},n) is selected in a way such that thenon-stationary pixels along the motion trajectory for sample C is thesame as that of its vertically neighboring pixels after video spatialup-conversion:

$\begin{matrix}{{\alpha_{C}( {\overset{arrow}{x},n} )} = {{CLIP}( {0,\frac{{{{\beta_{C\; 1} + \beta_{C\; 2}}}/2} + \delta}{{{{F_{init}( {\overset{arrow}{x},n} )} - {F_{SUC}( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} )}}} + \delta},1} )}} & (4)\end{matrix}$

In Equation (4), δ is a small constant preventing division by zero, and{right arrow over (μ)}_(y)=(1,0)^(T),β_(C1) =|F({right arrow over (x)}−{right arrow over (μ)} _(y) ,n)−F_(SUC)({right arrow over (x)}−{right arrow over (μ)} _(y) −{right arrowover (d)}({right arrow over (x)},n),n−1)|,β_(C2) =|F({right arrow over (x)}+{right arrow over (μ)} _(y) ,n)−F_(SUC)({right arrow over (x)}+{right arrow over (μ)} _(y) −{right arrowover (d)}({right arrow over (x)},n),n−1)|.

α_(D)({right arrow over (x)},n) is selected in a way such that thenon-stationary pixels along the motion trajectory for sample D is thesame as that of its four diagonally neighboring pixels after videospatial up-conversion:

$\begin{matrix}{{\alpha_{D}( {\overset{arrow}{x},n} )} = {{CLIP}( {0,\frac{{{{\beta_{D\; 1} + \beta_{D\; 2} + \beta_{D\; 3} + \beta_{D\; 4}}}/4} + \delta}{{{{F_{init}( {\overset{arrow}{x},n} )} - {F_{SUC}( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )}},{n - 1}} )}}} + \delta},1} )}} & (5)\end{matrix}$

In Equation (5), δ is a small constant preventing division by zero, andβ_(D1) =|F({right arrow over (x)}−{right arrow over (μ)} _(x)−{rightarrow over (μ)}_(y) ,n)−F _(SUC)({right arrow over (x)}−{right arrowover (μ)} _(x)−{right arrow over (μ)}_(y) −{right arrow over (d)}({rightarrow over (x)},n),n−1)|,β_(D2) =|F({right arrow over (x)}+{right arrow over (μ)} _(x)−{rightarrow over (μ)}_(y) ,n)−F _(SUC)({right arrow over (x)}+{right arrowover (μ)} _(x)−{right arrow over (μ)}_(y) −{right arrow over (d)}({rightarrow over (x)},n),n−1)|,β_(D3) =|F({right arrow over (x)}−{right arrow over (μ)} _(x)+{rightarrow over (μ)}_(y) ,n)−F _(SUC)({right arrow over (x)}−{right arrowover (μ)} _(x)+{right arrow over (μ)}_(y) −{right arrow over (d)}({rightarrow over (x)},n),n−1)|,β_(D4) =|F({right arrow over (x)}+{right arrow over (μ)} _(x)+{rightarrow over (μ)}_(y) ,n)−F _(SUC)({right arrow over (x)}+{right arrowover (μ)} _(x)+{right arrow over (μ)}_(y) −{right arrow over (d)}({rightarrow over (x)},n),n−1)|.

Algorithm II: Adaptively Recursive Video Spatial Up-Conversion Using aGeneralized Sampling Theorem (GST).

A continuous band-limited signal with a maximum frequency f_(max) can becompletely recovered from its discrete samples with a sampling frequencyof at least f_(s)=2f_(max). The generalized sampling theorem (GST),developed by Yen in 1956, has shown that any band-limited signal with amaximum frequency f_(max) can be completely recovered from its Ndisjoint sets of discrete samples, with each set obtained with asampling frequency of at least f_(s)=2f_(max)/N. In this situation, the“disjointness” refers to a shift in the time/spatial domain orequivalently, a phase difference in the frequency domain.

As shown in FIG. 6, where only the interpolation in the verticaldirection is considered, two disjoint sets of samples are available forthe interpolation of high-resolution samples in frame n, as long as thevertical component of the motion vector does not equal to 2k, k∈Z. Oneset is composed of the original samples in frame n, and the other set iscomposed of the motion compensated samples from frame (n−1). If themaximum vertical frequency satisfies f_(max) ^(y)≦2f_(s) ^(y)/2=f_(s)^(y), then the original continuous signal can be exactly recovered fromthe two sets of samples, and the interpolated samples can be furtherobtained by resampling of the reconstructed signal. This is the basicidea of using GST in interpolation.

GST has been successfully used in video deinterlacing. Extended from theknown deinterlacing algorithm Adaptive Recursive GST, the followingAdaptive Recursive Motion Compensated scheme using GST (ARMC-GST) can beused for video spatial up-conversion. ARMC-GST is a two-step algorithm.The first step involves horizontal interpolation. An Optimized FIRfilter is designed for the interpolation in the horizontal direction,which is a 1D interpolation problem that conventionally understood.Through horizontal interpolation, samples at B positions are obtained.The second step involves vertical interpolation. Vertical interpolationis implemented as follows to obtain samples at C and D positions:

$\begin{matrix}{{F_{SUC}( {\overset{arrow}{x},n} )} = \{ \begin{matrix}\begin{matrix}{{{\alpha_{C}( {\overset{arrow}{x},n} )}{F_{init}( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{C}( {\overset{arrow}{x},n} )}} )F_{GST}}} \\{( {\overset{arrow}{x},n} ),{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {0,1} )}}\end{matrix} \\\begin{matrix}{{{\alpha_{D}( {\overset{arrow}{x},n} )}{F_{init}( {\overset{arrow}{x},n} )}} + {( {1 - {\alpha_{D}( {\overset{arrow}{x},n} )}} )F_{GST}}} \\{( {\overset{arrow}{x},n} ),{( {{x\mspace{11mu}{mod}\mspace{11mu} 2},{y\mspace{11mu}{mod}\mspace{11mu} 2}} ) = ( {1,1} )}}\end{matrix}\end{matrix} } & (6)\end{matrix}$

In equation (6), α_(C)({right arrow over (x)},n) is obtained by Equation(4), α_(D)({right arrow over (x)},n) is obtained by Equation (5),F_(init)({right arrow over (x)},n) is obtained by any spatialinterpolation technique, and

$\begin{matrix}\begin{matrix}{{F_{GST}( {\overset{arrow}{x},n} )} = {{\sum\limits_{k}\;{{F( {{\overset{arrow}{x} - {( {{2k} + 1} ){\overset{arrow}{\mu}}_{y}}},n} )}{h_{1}( {k,{d_{y}( {\overset{arrow}{x},n} )}} )}}} +}} \\{\sum\limits_{m}\;{F_{SUC}( {{\overset{arrow}{x} - {\overset{arrow}{d}( {\overset{arrow}{x},n} )} - {( {{2m} + 1} ){\overset{arrow}{\mu}}_{y}}},{n - 1}} )}} \\{h_{2}( {m,{d_{y}( {\overset{arrow}{x},n} )}} )}\end{matrix} & (7)\end{matrix}$

In Equation (7), h₁(k,d_(y)({right arrow over (x)},n)) andh₂(m,d_(y)({right arrow over (x)},n)), k,m∈Z, are two FIR filters in thevertical direction as a function of the vertical component of the motionvector. The FIR filters can be designed in exactly the same way as theirdesign in video deinterlacing using GST. As designated by thereferences, it is known that, in recovering a continuous signal from itstwo sets of disjoint samples, if the condition f_(max) ^(y)≦2f_(s)^(y)/2=f_(s) ^(y) is satisfied, aliasing for each set of samples iscaused only by the interference of the two adjacent spectralreplications. The spectrum of the original continuous signal cantherefore be expressed in closed from as a linear combination of thespectrums of the two set of samples with complex weights. Theinterpolated samples are then obtained through resampling of thereconstructed continuous signal.

FIGS. 7 and 8 show one representative electronic device 12 upon whichthe present invention may be implemented. The electronic device 12 shownin FIGS. 7 and 8 comprises a mobile telephone. However, it is importantto note that the present invention is not limited to any type ofelectronic device and could be incorporated into devices such aspersonal digital assistants, personal computers, integrated messagingdevices, and a wide variety of other devices. It should be understoodthat the present invention could be incorporated on a wide variety ofelectronic device 12.

The electronic device 12 of FIGS. 7 and 8 includes a housing 30, adisplay 32 in the form of a liquid crystal display, a keypad 34, amicrophone 36, an ear-piece 38, a battery 40, an infrared port 42, anantenna 44, a smart card 46 in the form of a universal integratedcircuit card (UICC) according to one embodiment of the invention, a cardreader 48, radio interface circuitry 52, codec circuitry 54, acontroller 56 and a memory 58. It should be noted that the controller 56can be the same unit or a different unit than the camera processor 116.The memory 58 may or may not be the same component as the primary memoryunit 114 in various embodiments of the present invention. Individualcircuits and elements are all of a type well known in the art, forexample in the Nokia range of mobile telephones.

The present invention can be implemented as a part of a TV-out systemfor mobile terminals. Such a system can permit a user to display videoscaptured by a handset in a separate TV device.

The present invention is described in the general context of methodsteps, which may be implemented in one embodiment by a program productincluding computer-executable instructions, such as program code,executed by computers in networked environments.

Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of program code for executing steps of the methods disclosedherein. The particular sequence of such executable instructions orassociated data structures represents examples of corresponding acts forimplementing the functions described in such steps.

Software and web implementations of the present invention could beaccomplished with standard programming techniques with rule-based logicand other logic to accomplish the various database searching steps,correlation steps, comparison steps and decision steps. It should alsobe noted that the words “component” and “module” as used herein, and inthe claims, is intended to encompass implementations using one or morelines of software code, and/or hardware implementations, and/orequipment for receiving manual inputs.

The foregoing description of embodiments of the present invention havebeen presented for purposes of illustration and description. It is notintended to be exhaustive or to limit the present invention to theprecise form disclosed, and modifications and variations are possible inlight of the above teachings or may be acquired from practice of thepresent invention. The embodiments were chosen and described in order toexplain the principles of the present invention and its practicalapplication to enable one skilled in the art to utilize the presentinvention in various embodiments and with various modifications as aresuited to the particular use contemplated.

1. A method for performing motion compensated video spatialup-conversion on video comprising first and second fields, each of thefirst and second fields including a plurality of horizontal samples anda plurality of vertical samples, comprising: interpolating the pluralityof horizontal samples in the first field and the second field using aspatial interpolation technique; and interpolating the plurality ofvertical samples in the first field and the second field using a motioncompensated deinterlacing technique, the motion compensateddeinterlacing technique comprising at least one of an adaptivelyrecursive motion compensated video spatial up-conversion and anadaptively recursive video spatial up-conversion using a generalizedsampling theorem.
 2. The method of claim 1, wherein an optimized finiteimpulse response filter is used in the interpolation of the plurality ofhorizontal samples.
 3. The method of claim 1, wherein at least oneoptimized finite impulse response filter is used in the interpolation ofthe plurality of vertical samples.
 4. The method of claim 1, wherein thevideo is captured by a mobile device, and further comprising displayingthe video on a television after the plurality of horizontal samples andthe plurality of vertical samples have been interpolated.
 5. The methodof claim 1, wherein the video is intended for a non-high definitiontelevision, and further comprising displaying the video on a highdefinition television after the plurality of horizontal samples and theplurality of vertical samples have been interpolated.
 6. A computerprogram product for performing motion compensated video spatialup-conversion on video comprising first and second fields, each of thefirst and second fields including a plurality of horizontal samples anda plurality of vertical samples, comprising: computer code forinterpolating the plurality of horizontal samples in the first field andthe second field using a spatial interpolation technique; and computercode for interpolating the plurality of vertical samples in the firstfield and the second field using a motion compensated deinterlacingtechnique, the motion compensated deinterlacing technique comprising atleast one of an adaptively recursive motion compensated video spatialup-conversion and an adaptively recursive video spatial up-conversionusing a generalized sampling theorem.
 7. The computer program product ofclaim 6, wherein an optimized finite impulse response filter is used inthe interpolation of the plurality of horizontal samples.
 8. Thecomputer program product of claim 6, wherein at least one optimizedfinite impulse response filter is used in the interpolation of theplurality of vertical samples.
 9. The computer program product of claim6, wherein the video is captured by a mobile device, and furthercomprising computer code for displaying the video on a television afterthe plurality of horizontal samples and the plurality of verticalsamples have been interpolated.
 10. The computer program product ofclaim 6, wherein the video is intended for a non-high definitiontelevision, and further comprising displaying the video on a highdefinition television after the plurality of horizontal samples and theplurality of vertical samples have been interpolated.
 11. An electronicdevice, comprising: a processor; and a memory unit operatively connectedto the processor and including a computer program product for performingmotion compensated video spatial up-conversion on video comprising firstand second fields, each of the first and second fields including aplurality of horizontal samples and a plurality of vertical samples, thecomputer program product comprising: computer code for interpolating theplurality of horizontal samples in the first field and the second fieldusing a spatial interpolation technique; and computer code forinterpolating the plurality of vertical samples in the first field andthe second field using a motion compensated deinterlacing technique, themotion compensated deinterlacing technique comprising at least one of anadaptively recursive motion compensated video spatial up-conversion andan adaptively recursive video spatial up-conversion using a generalizedsampling theorem.
 12. The electronic device of claim 11, wherein anoptimized finite impulse response filter is used in the interpolation ofthe plurality of horizontal samples.
 13. The electronic device of claim11, wherein at least one optimized finite impulse response filter isused in the interpolation of the plurality of vertical samples.
 14. Theelectronic device of claim 11, wherein the electronic device comprises amobile telephone.